Signaling non-scalable-nested hypothetical reference video decoder information

ABSTRACT

Examples of video encoding methods and apparatus and video decoding methods and apparatus are described. An example method of video processing includes performing a conversion between a video and a bitstream of the video, wherein the bitstream includes one or more output layer sets comprising one or more video layers according to a format rule, wherein the format rule specifies that a non-scalable-nested supplemental enhancement information, SEI, message that includes information regarding hypothetical reference decoder, HRD, is applicable to all output layer sets that include same video layers as the bitstream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/US2021/036484, filed on Jun. 8, 2021, which claims the priorityto and benefits of U.S. Provisional Patent Application No. 63/036,808,filed on Jun. 9, 2020. All the aforementioned patent applications arehereby incorporated by reference in their entireties.

TECHNICAL FIELD

This patent document relates to image and video coding and decoding.

BACKGROUND

Digital video accounts for the largest bandwidth use on the internet andother digital communication networks. As the number of connected userdevices capable of receiving and displaying video increases, it isexpected that the bandwidth demand for digital video usage will continueto grow.

SUMMARY

The present document discloses techniques that can be used by videoencoders and decoders to perform video encoding or decoding.

In one example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video and a bitstreamof the video, wherein the bitstream includes one or more output layersets comprising one or more video layers according to a format rule,wherein the format rule specifies that a non-scalable-nestedsupplemental enhancement information, SEI, message that includesinformation regarding hypothetical reference decoder, HRD, is applicableto all output layer sets that include same video layers as thebitstream.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that anon-scalable-nested hypothetical reference decoder (HRD)-relatedsupplemental enhancement information (SEI) message is omitted responsiveto a condition that there does not exist an output layer set thatincludes a same set of layers as the bitstream.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that a value of layeridentifier for a supplemental enhancement information (SEI) networkabstraction layer (NAL) unit that includes a non-scalable-nested SEImessage is not constrained.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that a particular payloadtype value corresponding to subpicture level information is disallowedfrom a list that includes allowable supplemental enhancement information(SEI) payload type values for non-hypothetical reference decoder (HRD)related supplemental enhancement information (SEI) messages.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream includes one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that anon-scalable-nested supplemental enhancement information, SEI, messagethat includes information irrelevant to hypothetical reference decoder(HRD) is applicable to all layers in the bitstream.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video comprising one or more output layer setsaccording to a rule, wherein the rule specifies that a supplementalenhancement information (SEI) network abstraction layer (NAL) unit thatincludes a scalable-nested SEI message carrying picture timinginformation is not included due to use of a same picture timing in alloutput layer sets in the bitstream.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moresupplemental enhancement information, SEI, network abstraction layer(NAL) units according to a rule, wherein the rule specifies that,responsive to an SEI NAL unit including a non-scalable-nested SEImessage of a first payload type, the SEI NAL unit is disallowed toinclude another SEI message of a second payload type.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video according to a rule, wherein the bitstreamcomprises one or more supplemental enhancement information, SEI, networkabstraction layer (NAL) units according to a rule, wherein the rulespecifies that, responsive to an SEI NAL unit including ascalable-nested SEI message of a first payload type, the SEI NAL unit isdisallowed to include another SEI message of a second payload type.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moresupplemental enhancement information, SEI, network abstraction layer(NAL) units according to a rule, wherein the rule specifies that,responsive to an SEI NAL unit including a SEI message of a first payloadtype, the SEI NAL unit is disallowed to include another SEI message notequal to the first payload type or a second payload type.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to arule, wherein the rule specifies a particular decoding order between asubpicture level information (SLI) supplemental enhancement information(SEI) message and a buffering period (BP) SEI message that apply to aparticular output layer set, responsive to a condition that the SLI SEImessage and the BP SEI message are included in an access unit.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video and abitstream of the video according to a rule, wherein the rule specifiesthat a first syntax field indicating sublayer representation informationfor which an initial coded picture buffer (CPB) removal delay relatedsyntax elements are present is omitted, responsive to a particular valueof a second syntax field indicative of a maximum number of temporalsublayers for which an initial CPB removal delay is indicated in abuffering period supplemental enhancement information (SEI) message.

In yet another example aspect, a video encoder apparatus is disclosed.The video encoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a video decoder apparatus is disclosed.The video decoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a computer readable medium having codestored thereon is disclose. The code embodies one of the methodsdescribed herein in the form of processor-executable code.

These, and other, features are described throughout the presentdocument.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram that illustrates a video coding system inaccordance with some implementations of the disclosed technology.

FIG. 2 is a block diagram of an example hardware platform used for videoprocessing.

FIG. 3 is a flowchart for an example method of video processing.

FIG. 4 is a block diagram that illustrates an example video codingsystem.

FIG. 5 is a block diagram that illustrates an encoder in accordance withsome implementations of the disclosed technology.

FIG. 6 is a block diagram that illustrates a decoder in accordance withsome implementations of the disclosed technology.

FIGS. 7A to 7E are flowcharts for example methods of video processingbased on some implementations of the disclosed technology.

FIG. 8 is a flowchart for an example method of video processing based onsome implementations of the disclosed technology.

FIGS. 9A to 9C are flowcharts for example methods of video processingbased on some implementations of the disclosed technology.

FIGS. 10A and 10B are flowcharts for example methods of video processingbased on some implementations of the disclosed technology.

DETAILED DESCRIPTION

Section headings are used in the present document for ease ofunderstanding and do not limit the applicability of techniques andembodiments disclosed in each section only to that section. Furthermore,H.266 terminology is used in some description only for ease ofunderstanding and not for limiting scope of the disclosed techniques. Assuch, the techniques described herein are applicable to other videocodec protocols and designs also.

1. Introduction

This document is related to video coding technologies. Specifically, itis about defining levels and bitstream conformance for a video codecthat supports both single-layer video coding and multi-layer videocoding. It may be applied to any video coding standard or non-standardvideo codec that supports single-layer video coding and multi-layervideo coding, e.g., Versatile Video Coding (VVC) that is beingdeveloped.

2. Abbreviations

-   -   APS Adaptation Parameter Set    -   AU Access Unit    -   AUD Access Unit Delimiter    -   AVC Advanced Video Coding    -   BP Buffering Period    -   CLVS Coded Layer Video Sequence    -   CPB Coded Picture Buffer    -   CRA Clean Random Access    -   CTU Coding Tree Unit    -   CVS Coded Video Sequence    -   DPB Decoded Picture Buffer    -   DPS Decoding Parameter Set    -   DUI Decoding Unit Information    -   EOB End Of Bitstream    -   EOS End Of Sequence    -   GCI General Constraints Information    -   GDR Gradual Decoding Refresh    -   HEVC High Efficiency Video Coding    -   HRD Hypothetical Reference Decoder    -   IDR Instantaneous Decoding Refresh    -   JEM Joint Exploration Model    -   MCTS Motion-Constrained Tile Sets    -   NAL Network Abstraction Layer    -   OLS Output Layer Set    -   PH Picture Header    -   PPS Picture Parameter Set    -   PT Picture Timing    -   PTL Profile, Tier and Level    -   PU Picture Unit    -   RRP Reference Picture Resampling    -   RB SP Raw Byte Sequence Payload    -   SEI Supplemental Enhancement Information    -   SH Slice Header    -   SLI Subpicture Level Information    -   SPS Sequence Parameter Set    -   SVC Scalable Video Coding    -   VCL Video Coding Layer    -   VPS Video Parameter Set    -   VTM VVC Test Model    -   VUI Video Usability Information    -   VVC Versatile Video Coding

3. Initial Discussion

Video coding standards have evolved primarily through the development ofthe well-known ITU-T and ISO/IEC standards. The ITU-T produced H.261 andH.263, ISO/IEC produced MPEG-1 and MPEG-4 Visual, and the twoorganizations jointly produced the H.262/MPEG-2 Video and H.264/MPEG-4Advanced Video Coding (AVC) and H.265/HEVC standards. Since H.262, thevideo coding standards are based on the hybrid video coding structurewherein temporal prediction plus transform coding are utilized. Toexplore the future video coding technologies beyond HEVC, the JointVideo Exploration Team (WET) was founded by VCEG and MPEG jointly in2015. Since then, many new methods have been adopted by JVET and putinto the reference software named Joint Exploration Model (JEM). TheJVET meeting is concurrently held once every quarter, and the new codingstandard is targeting at 50% bitrate reduction as compared to HEVC. Thenew video coding standard was officially named as Versatile Video Coding(VVC) in the April 2018 JVET meeting, and the first version of VVC testmodel (VTM) was released at that time. As there are continuous effortcontributing to VVC standardization, new coding techniques are beingadopted to the VVC standard in every JVET meeting. The VVC working draftand test model VTM are then updated after every meeting. The VVC projectis now aiming for technical completion (FDIS) at the July 2020 meeting.

3.1. Parameter Sets

AVC, HEVC, and VVC specify parameter sets. The types of parameter setsinclude SPS, PPS, APS, and VPS. SPS and PPS are supported in all of AVC,HEVC, and VVC. VPS was introduced since HEVC and is included in bothHEVC and VVC. APS was not included in AVC or HEVC but is included in thelatest VVC draft text.

SPS was designed to carry sequence-level header information, and PPS wasdesigned to carry infrequently changing picture-level headerinformation. With SPS and PPS, infrequently changing information neednot to be repeated for each sequence or picture, hence redundantsignaling of this information can be avoided. Furthermore, the use ofSPS and PPS enables out-of-band transmission of the important headerinformation, thus not only avoiding the need for redundant transmissionsbut also improving error resilience.

VPS was introduced for carrying sequence-level header information thatis common for all layers in multi-layer bitstreams.

APS was introduced for carrying such picture-level or slice-levelinformation that needs quite some bits to code, can be shared bymultiple pictures, and in a sequence there can be quite many differentvariations.

3.2. Picture Resolution Change within a Sequence

In AVC and HEVC, the spatial resolution of pictures cannot change unlessa new sequence using a new SPS starts, with an TRAP picture. VVC enablespicture resolution change within a sequence at a position withoutencoding an TRAP picture, which is always intra-coded. This feature issometimes referred to as reference picture resampling (RPR), as thefeature needs resampling of a reference picture used for interprediction when that reference picture has a different resolution thanthe current picture being decoded.

The scaling ratio is restricted to be larger than or equal to 1/2 (2times downsampling from the reference picture to the current picture),and less than or equal to 8 (8 times upsampling). Three sets ofresampling filters with different frequency cutoffs are specified tohandle various scaling ratios between a reference picture and thecurrent picture. The three sets of resampling filters are appliedrespectively for the scaling ratio ranging from 1/2 to 1/1.75, from1/1.75 to 1/1.25, and from 1/1.25 to 8. Each set of resampling filtershas 16 phases for luma and 32 phases for chroma which is same to thecase of motion compensation interpolation filters. Actually the normalMC interpolation process is a special case of the resampling processwith scaling ratio ranging from 1/1.25 to 8. The horizontal and verticalscaling ratios are derived based on picture width and height, and theleft, right, top and bottom scaling offsets specified for the referencepicture and the current picture.

Other aspects of the VVC design for support of this feature that aredifferent from HEVC include: i) The picture resolution and thecorresponding conformance window are signaled in the PPS instead of inthe SPS, while in the SPS the maximum picture resolution is signaled.ii) For a single-layer bitstream, each picture store (a slot in the DPBfor storage of one decoded picture) occupies the buffer size as requiredfor storing a decoded picture having the maximum picture resolution.

3.3. Scalable Video Coding (SVC) in General and in VVC

Scalable video coding (SVC, sometimes also just referred to asscalability in video coding) refers to video coding in which a baselayer (BL), sometimes referred to as a reference layer (RL), and one ormore scalable enhancement layers (ELs) are used. In SVC, the base layercan carry video data with a base level of quality. The one or moreenhancement layers can carry additional video data to support, forexample, higher spatial, temporal, and/or signal-to-noise (SNR) levels.Enhancement layers may be defined relative to a previously encodedlayer. For example, a bottom layer may serve as a BL, while a top layermay serve as an EL. Middle layers may serve as either ELs or RLs, orboth. For example, a middle layer (e.g., a layer that is neither thelowest layer nor the highest layer) may be an EL for the layers belowthe middle layer, such as the base layer or any intervening enhancementlayers, and at the same time serve as a RL for one or more enhancementlayers above the middle layer. Similarly, in the Multiview or 3Dextension of the HEVC standard, there may be multiple views, andinformation of one view may be utilized to code (e.g., encode or decode)the information of another view (e.g., motion estimation, motion vectorprediction and/or other redundancies).

In SVC, the parameters used by the encoder or the decoder are groupedinto parameter sets based on the coding level (e.g., video-level,sequence-level, picture-level, slice level, etc.) in which they may beutilized. For example, parameters that may be utilized by one or morecoded video sequences of different layers in the bitstream may beincluded in a video parameter set (VPS), and parameters that areutilized by one or more pictures in a coded video sequence may beincluded in a sequence parameter set (SPS). Similarly, parameters thatare utilized by one or more slices in a picture may be included in apicture parameter set (PPS), and other parameters that are specific to asingle slice may be included in a slice header. Similarly, theindication of which parameter set(s) a particular layer is using at agiven time may be provided at various coding levels.

Thanks to the support of reference picture resampling (RPR) in VVC,support of a bitstream containing multiple layers, e.g., two layers withSD and HD resolutions in VVC can be designed without the need anyadditional signal-processing-level coding tool, as upsampling needed forspatial scalability support can just use the RPR upsampling filter.Nevertheless, high-level syntax changes (compared to not supportingscalability) are needed for scalability support. Scalability support isspecified in VVC version 1. Different from the scalability supports inany earlier video coding standards, including in extensions of AVC andHEVC, the design of VVC scalability has been made friendly tosingle-layer decoder designs as much as possible. The decodingcapability for multi-layer bitstreams are specified in a manner as ifthere were only a single layer in the bitstream. E.g., the decodingcapability, such as DPB size, is specified in a manner that isindependent of the number of layers in the bitstream to be decoded.Basically, a decoder designed for single-layer bitstreams does not needmuch change to be able to decode multi-layer bitstreams. Compared to thedesigns of multi-layer extensions of AVC and HEVC, the HLS aspects havebeen significantly simplified at the sacrifice of some flexibilities.For example, an TRAP AU is required to contain a picture for each of thelayers present in the CVS.

3.4. SEI Messages and General SEI Semantics and Constraints

Annex D of VVC specifies syntax and semantics for SEI message payloadsfor some SEI messages, and specifies the use of the SEI messages and VUIparameters for which the syntax and semantics are specified in ITU-TH.SEI|ISO/IEC 23002-7.

SEI messages assist in processes related to decoding, display or otherpurposes. However, SEI messages are not required for constructing theluma or chroma samples by the decoding process. Conforming decoders arenot required to process this information for output order conformance.Some SEI messages are required for checking bitstream conformance andfor output timing decoder conformance. Other SEI messages are notrequired for check bitstream conformance.

In the latest VVC draft text, the general SEI.

The syntax and semantics of the scalable nesting SEI message in thelatest VVC draft text are as follows.

D.2.1 General SEI Message Syntax

sei_payload( payloadType, payloadSize ) { Descriptor  if( nal_unit_type== PREFIX_SEI_NUT )   if( payloadType = = 0 )    buffering_period(payloadSize )   else if( payloadType = = 1 )    pic_timing( payloadSize)   else if( payloadType = = 3 )    filler_payload( payloadSize ) /*Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   else if( payloadType = =4 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   user_data_registered_itu_t_t35( payloadSize )   else if( payloadType= = 5 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   user_data_unregistered( payloadSize )   else if( payloadType = = 19 )/* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   film_grain_characteristics( payloadSize )   else if( payloadType = =45 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   frame_packing_arrangement( payloadSize )   else if( payloadType = =129 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   parameter_sets_inclusion_indication( payloadSize )   else if(payloadType = = 130 )    decoding_unit_info( payloadSize )   else if(payloadType = = 133 )    scalable_nesting( payloadSize )   else if(payloadType = = 137 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   mastering_display_colour_volume( payloadSize )   else if( payloadType= = 144 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   content_light_level_info( payloadSize )   else if( payloadType = =145 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   dependent_rap_indication( payloadSize )   else if( payloadType = =147 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   alternative_transfer_characteristics( payloadSize )   else if(payloadType = = 148 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   ambient_viewing_environment( payloadSize )   else if( payloadType = =149 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   content_colour_volume( payloadSize )   else if( payloadType = = 150 )/* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   equirectangular_projection( payloadSize )   else if( payloadType = =153 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   generalized_cubemap_projection( payloadSize )   else if( payloadType= = 154 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */    sphererotation/ payloadSize )   else if( payloadType = = 155 ) /* Specified inITU-T H.SEI | ISO/IEC 23002-7 */    regionwise_packing( payloadSize )  else if( payloadType = = 156 ) /* Specified in ITU-T H.SEI | ISO/IEC23002-7 */    omni_viewport( payloadSize )   else if( payloadType = =168 ) /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   frame_field_info( payloadSize )   else if( payloadType = = 203 )   subpic_level_info( payloadSize )   else if( payloadType = = 204 ) /*Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   sample_aspect_ratio_info( payloadSize )   else           /* Specifiedin ITU-T H. SEI | ISO/IEC 23002-7 */    reserved_message( payloadSize ) else /* nal_unit_type = = SUFFIX_SEI_NUT */   if( payloadType = = 3 )/* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */    filler_payload(payloadSize )   if( payloadType = = 132 ) /* Specified in ITU-T H.SEI |ISO/IEC 23002-7 */    decoded_picture_hasli( payloadSize )   else if(payloadType = = 133 )    scalable_nesting( payloadSize )  else         /* Specified in ITU-T H.SEI | ISO/IEC 23002-7 */   reserved_message( payloadSize )  if( more_data_in_payload( ) ) {  if( payload_extension_present( ) )    reservedpayloadextensiondata u(v)   payload bit equal to one /* equal to 1 */ f(1 )   while/!byte_aligned( ) )    payload bit equal to zero /* equal to 0 */ f(1 ) } }

D.2.2 General SEI Payload Semantics

reserved_payload_extension_data shall not be present in bitstreamsconforming to this version of this Specification. However, decodersconforming to this version of this Specification shall ignore thepresence and value of reserved_payload_extension_data. When present, thelength, in bits, of reserved_payload_extension_data is equal to8*payloadSize—nEarlierBits—nPayloadZeroBits—1, where nEarlierBits is thenumber of bits in the sei_payload( ) syntax structure that precede thereserved_payload_extension_data syntax element, and nPayloadZeroBits isthe number of payload_bit_equal_to_zero syntax elements at the end ofthe sei_payload( ) syntax structure.payload_bit_equal_to_one shall be equal to 1.payload_bit_equal_to_zero shall be equal to 0.

-   -   NOTE 1—SEI messages with the same value of payloadType are        conceptually the same SEI message regardless of whether they are        contained in prefix or suffix SEI NAL units.    -   NOTE 2—For SEI messages with payloadType in the range of 0 to        47, inclusive, that are specified in this Specification, the        payloadType values are aligned with similar SEI messages        specified in Rec. ITU-T H.2641 ISO/IEC 14496-10.        The semantics and persistence scope for each SEI message are        specified in the semantics specification for each particular SEI        message.    -   NOTE 3—Persistence information for SEI messages is informatively        summarized in Table D.1.

TABLE D.1 Persistence scope of SEI messages (informative ) SEI messagePersistence scope Buffering period The remainder of the bitstreamPicture timing The AU containing the SEI message DU information The AUcontaining the SEI message Scalable nesting Depending on thescalable-nested SEI messages. Each scalable-nested SEI message has thesame persistence scope as if the SEI message was not scalable-nestedSubpicture level information The CLVS containing the SEI messageThe list VclAssociatedSeiList is set to consist of the payloadTypevalues 3, 19, 45, 129, 132, 137, 144, 145, 147 to 150, inclusive, 153 to156, inclusive, 168, 203, and 204.The list PicUnitRepConSeiList is set to consist of the payloadTypevalues 0, 1, 19, 45, 129, 132, 133, 137, 147 to 150, inclusive, 153 to156, inclusive, 168, 203, and 204.

-   -   NOTE 4—VclAssociatedSeiList consists of the payloadType values        of the SEI messages that, when non-scalable-nested and contained        in an SEI NAL unit, infer constraints on the NAL unit header of        the SEI NAL unit on the basis of the NAL unit header of the        associated VCL NAL unit. PicUnitRepConSeiList consists of the        payloadType values of the SEI messages that are subject to the        restriction on 4 repetitions per PU.        It is a requirement of bitstream conformance that the following        restrictions apply on containing of SEI messages in SEI NAL        units:    -   When an SEI NAL unit contains a non-scalable-nested BP SEI        message, a non-scalable-nested PT SEI message, or a        non-scalable-nested DUI SEI message, the SEI NAL unit shall not        contain any other SEI message with payloadType not equal to 0        (BP), 1 (PT), or 130 (DUI).    -   When an SEI NAL unit contains a scalable-nested BP SEI message,        a scalable-nested PT SEI message, or a scalable-nested DUI SEI        message, the SEI NAL unit shall not contain any other SEI        message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI)        or 133 (scalable nesting).        The following applies on the applicable OLSs or layers of        non-scalable-nested SEI messages:    -   For a non-scalable-nested SEI message, when payloadType is equal        to 0 (BP), 1 (PT), or 130 (DUI), the non-scalable-nested SEI        message applies only to the 0-th OLS.    -   For a non-scalable-nested SEI message, when payloadType is equal        to any value among VclAssociatedSeiList, the non-scalable-nested        SEI message applies only to the layer for which the VCL NAL        units have nuh_layer_id equal to the nuh_layer_id of the SEI NAL        unit containing the SEI message.        It is a requirement of bitstream conformance that the following        restrictions apply on the value of nuh_layer_id of SEI NAL        units:    -   When a non-scalable-nested SEI message has payloadType equal to        0 (BP), 1 (PT), or 130 (DUI), the SEI NAL unit containing the        non-scalable-nested SEI message shall have nuh_layer_id equal to        vps_layer_id[0].    -   When a non-scalable-nested SEI message has payloadType equal to        any value among VclAssociatedSeiList, the SEI NAL unit        containing the non-scalable-nested SEI message shall have        nuh_layer_id equal to the value of nuh_layer_id of the VCL NAL        unit associated with the SEI NAL unit.    -   An SEI NAL unit containing a scalable nesting SEI message shall        have nuh_layer_id equal to the lowest value of nuh_layer_id of        all layers to which the scalable-nested SEI messages apply (when        sn_ols_flag of the scalable nesting SEI message is equal to 0)        or the lowest value of nuh_layer_id of all layers in the OLSs to        which the scalable-nested SEI message apply (when sn_ols_flag of        the scalable nesting SEI message is equal to 1).        It is a requirement of bitstream conformance that the following        restrictions apply on repetition of SEI messages:    -   For each of the payloadType values included in        PicUnitRepConSeiList, there shall be less than or equal to 4        identical sei_payload( ) syntax structures within a PU.    -   There shall be less than or equal to 4 identical sei_payload( )        syntax structures with payloadType equal to 130 within a DU.        The following applies on the order of BP, PT, and DUI SEI        messages:    -   When a BP SEI message and a PT SEI message that apply to a        particular OP are present within an AU, the BP SEI messages        shall precede the PT SEI message in decoding order.    -   When a BP SEI message and a DUI SEI message that apply to a        particular OP are present within an AU, the BP SEI messages        shall precede the DUI SEI message in decoding order.    -   When a PT SEI message and a DUI SEI message that apply to a        particular OP are present within an AU, the PT SEI messages        shall precede the DUI SEI message in decoding order.

4. Technical Problems Solved by Disclosed Technical Solutions

The existing general SEI payload semantics, including the general SEIconstraints, has the following problems:

-   1) Non-scalable-nested HRD-related SEI messages should be specified    to apply to the OLSs that include the same set of layers as the    entire bitstream (instead of apply only to the 0th OLS).-   2) An entire bitstream may include multiple layers while there is no    OLS specified that includes all the layer. In this case, there    cannot be any non-scalable-nested HRD-related SEI message, as they    would apply to the OLSs that include the same set of layers as the    entire bitstream.-   3) It is currently specified that a non-scalable-nested    non-HRD-related SEI message only applies to the layer with    nuh_layer_id equal to that of the SEI NAL unit. However, to be    consistent with non-scalable-nested HRD-related SEI messages, it    should be specified that non-scalable-nested non-HRD-related SEI    messages apply to all layers in the entire bitstream.-   4) It is currently specified that the value of nuh_layer_id for an    SEI NAL unit containing non-scalable-nested HRD-related SEI messages    shall be equal to vps_layer_id[0] and the value of nuh_layer_id for    an SEI NAL unit containing non-scalable-nested non-HRD-related SEI    messages shall be equal to the nuh_layer_id of the VCL NAL unit    associated with the SEI NAL unit. However, as non-scalable-nested    SEI messages apply to the entire bitstream, these constraints on the    value of nuh_layer_id should be removed, such that the value of    nuh_layer_id for SEI NAL units containing non-scalable-nested SEI    messages is unconstrained, same as for the nuh_layer_id for DCI,    VPS, AUD, and EOS NAL units.-   5) The list variable VclAssociatedSeiList currently consists of the    SEI payloadType values for non-HRD-related SEI messages. However,    the value 203, for the SLI SEI message, is also an HRD-related SEI    message. Therefore, the payloadType value 203 (SLI) should be    removed from the list.-   6) There lacks a constraint such that when general    same_pic_timing_in_all_ols_flag is equal to 1, there shall be no SEI    NAL unit that contain a scalable-nested SEI message with payloadType    equal to 1 (PT). This is because when general    same_pic_timing_in_all_ols_flag is equal to 1, there is no need to    have PT SEI messages contained in scalable nesting SEI messages.-   7) There lacks a constraint such that when an SEI NAL unit contains    a non-scalable-nested SEI message with payloadType equal to 0 (BP),    1 (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain    any other SEI message with payloadType not equal to 0, 1, 130,    or 203. Only with this, removal of these four HRD-related SEI    messages from the output bitstream during a sub-bitstream extraction    process can be performed by simply removing SEI NAL units containing    one or more of these SEI messages.-   8) There lacks a constraint such that when an SEI NAL unit contains    a scalable-nested SEI message with payloadType equal to 0 (BP), 1    (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit shall not contain    any other SEI message with payloadType not equal to 0, 1, 130, 203,    or 133 (scalable nesting). Only with this, it is possible to set the    value of sn_ols_flag of scalable nesting SEI messages for    scalable-nested HRD-related and non-HRD-related SEI messages without    a problem.-   9) There lacks a constraint such that when an SEI NAL unit contains    an SEI message with payloadType equal to 3 (filler payload), the SEI    NAL unit shall not contain any other SEI message with payloadType    not equal to 3. Only with this, removal of filler payload SEI    messages from the output bitstream during a sub-bitstream extraction    process can be performed by simply removing SEI NAL units containing    one or more filler payload SEI messages.-   10) There lacks a constraint such that when an SLI SEI message and a    BP SEI message that apply to a particular OLS are present within an    AU, the SLI SEI messages shall precede the BP SEI message in    decoding order. This is needed because an SLI SEI message provides    sequence level information like VPSs and SPSs, which also precede BP    SEI messages, when present within the same AU as the BP SEI    messages.-   11) In the BP SEI message, the    bp_sublayer_initial_cpb_removal_delay_present_flag is signalled even    when bp_max_sublayers_minus1 is equal to 0. However, when    bp_max_sublayers_minus1 is equal to 0, the value of    bp_sublayer_initial_cpb_removal_delay_present_flag is known to be 0.

5. Examples of Solutions and Embodiments

To solve the above problems, and others, methods as summarized below aredisclosed. The solution items should be considered as examples toexplain the general concepts and should not be interpreted in a narrowway. Furthermore, these items can be applied individually or combined inany manner.

-   1) To solve the 1st problem, specify that non-scalable-nested    HRD-related SEI messages apply to the OLSs that include the same set    of layers as the entire bitstream (instead of apply only to the 0th    OLS).    -   a. In one example, HRD-related SEI messages refer to SEI        messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or        203 (SLI).-   2) To solve the 2nd problem, add a constraint such that when there    is no OLS that includes the set of layers same as the entire    bitstream, there shall be no non-scalable-nested HRD-related SEI    messages.    -   a. In one example, HRD-related SEI messages refer to SEI        messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or        203 (SLI).-   3) To solve the 3rd problem, specify that non-scalable-nested    non-HRD-related SEI messages are applicable to all layers in the    entire bitstream, to be consistent with non-scalable-nested    HRD-related SEI messages.    -   a. In one example, non-HRD-related SEI messages refer to SEI        messages with payloadType not equal to 0 (BP), 1 (PT), 130        (DUI), or 203 (SLI).-   4) To solve the 4th problem, remove the constraints on the value of    nuh_layer_id for SEI NAL units containing non-scalable-nested SEI    messages, such that the value of nuh_layer_id for SEI NAL units    containing non-scalable-nested SEI messages is unconstrained, same    as for the nuh_layer_id for DCI, VPS, AUD, and EOS NAL units.-   5) To solve the 5th problem, rename VclAssociatedSeiList to    NestingForLayersSeiList, and remove payloadType value 203 (SLI) from    the list.-   6) To solve the 6th problem, add a constraint such that when general    same_pic_timing_in_all_ols_flag is equal to 1, there shall be no SEI    NAL unit that contain a scalable-nested SEI message with payloadType    equal to 1 (PT).    -   a. In one example, additionally, it is specified that when        general same_pic_timing_in_all_ols_flag is equal to 1,        non-scalable-nested PT SEI messages apply to all OLSs as well as        to the subpicture sequences that can be extracted from the        bitstreams of the OLSs.    -   b. In one example, alternatively, add a constraint such that        when general same_pic_timing_in_all_ols_flag is equal to 1,        there shall be no SEI NAL unit that contains a scalable-nested        SEI message with payloadType equal to 1 (PT) for which        sn_subpic_flag is equal to 0.-   7) To solve the 7th problem, it is specified that when an SEI NAL    unit contains a non-scalable-nested SEI message with payloadType    equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit    shall not contain any other SEI message with payloadType not equal    to 0, 1, 130, or 203.-   8) To solve the 8th problem, it is specified that when an SEI NAL    unit contains a scalable-nested SEI message with payloadType equal    to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI), the SEI NAL unit shall    not contain any other SEI message with payloadType not equal to 0,    1, 130, 203, or 133 (scalable nesting).-   9) To solve the 9th problem, add a constraint such that when an SEI    NAL unit contains an SEI message with payloadType equal to 3 (filler    payload), the SEI NAL unit shall not contain any other SEI message    with payloadType not equal to 3.    -   a. In one example, additionally, it is specified that filler        data SEI messages shall not be scalable-nested, i.e., shall not        be contained in a scalable nesting SEI message.    -   b. In one example, alternatively, add a constraint such that        when an SEI NAL unit contains an SEI message with payloadType        equal to 3 (filler payload), the SEI NAL unit shall not contain        any other SEI message with payloadType not equal to 3 or 133        (scalable nesting).-   10) To solve the 10th problem, add a constraint such that when an    SLI SEI message and a BP SEI message that apply to a particular OLS    are present within an AU, the SLI SEI messages shall precede the BP    SEI message in decoding order.-   11) To solve the 11th problem, it is specified that, when    bp_max_sublayers_minus1 is equal to 0,    bp_sublayer_initial_cpb_removal_delay_present_flag is skipped (i.e.,    not signalled in the BP SEI message).    -   a. In one example, additionally, when bp_max_sublayers_minus1 is        equal to 0, the value of        bp_sublayer_initial_cpb_removal_delay_present_flag is inferred        to be equal to 0.

6. EMBODIMENTS

Below are some example embodiments for some of the invention aspectssummarized above in this Section, which can be applied to the VVCspecification. The changed texts are based on the latest VVC text inJVET-S0152-v5. Most relevant parts that have been added or modified arebold and Italic, and some of the deleted parts are marked with doublebrackets (e.g., [[a]] denotes the deletion of the character “a”).

6.1. Embodiment 1

This embodiment is for items 1 to 11 and some of their sub-items.

D.2.2 General SEI Payload Semantics

The list NestingForLayersSeiList is set to consist of the payloadTypevalues 3, 19, 45, 129, 132, 137, 144, 145, 147 to 150, inclusive, 153 to156, inclusive, 168, and 204.The list PicUnitRepConSeiList is set to consist of the payloadTypevalues 0, 1, 19, 45, 129, 132, 133, 137, 147 to 150, inclusive, 153 to156, inclusive, 168, 203, and 204.

-   -   NOTE 4—NestingForLayersSeiList consists of the payloadType        values of the non-HRD-related SEI messages, for which, when        scalable-nested, the value of sn_ols_flag of the containing        scalable nesting SEI message shall be equal to 1.        PicUnitRepConSeiList consists of the payloadType values of the        SEI messages that are subject to the restriction on 4        repetitions per PU.        It is a requirement of bitstream conformance that the following        restrictions apply on containing of SEI messages in SEI NAL        units:    -   When general same_pic_timing_in_all_ols_flag is equal to 1,        there shall be no SEI NAL unit that contain a scalable-nested        SEI message with payloadType equal to 1 (PT).    -   When an SEI NAL unit contains a non-scalable-nested SEI message        with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203        (SLI), the SEI NAL unit shall not contain any other SEI message        with payloadType not equal to 0, 1, 130, or 203.    -   When an SEI NAL unit contains a scalable-nested SEI message with        payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI),        the SEI NAL unit shall not contain any other SEI message with        payloadType not equal to 0, 1, 130, 203, or 133 (scalable        nesting).    -   When an SEI NAL unit contains an SEI message with payloadType        equal to 3 (filler payload), the SEI NAL unit shall not contain        any other SEI message with payloadType not equal to 3.        The following applies on the applicable OLSs or layers of        non-scalable-nested SEI messages:    -   For a non-scalable-nested SEI message, when payloadType is equal        to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI), the        non-scalable-nested SEI message applies to the OLSs, when        present, that include the same set of layers as in the entire        bitstream. When there is no OLS that includes the same set of        layers as the entire bitstream, there shall be no        non-scalable-nested SEI message with payloadType equal to 0        (BP), 1 (PT), 130 (DUI), or 203 (SLI).    -   For a non-scalable-nested SEI message, when payloadType is equal        to any value among NestingForLayersSeiList, the        non-scalable-nested SEI message applies to all layers in the        entire bitstream.        It is a requirement of bitstream conformance that the following        restrictions apply on the value of nuh_layer_id of SEI NAL        units:    -   An SEI NAL unit containing a scalable nesting SEI message shall        have nuh_layer_id equal to the lowest value of nuh_layer_id of        all layers to which the scalable-nested SEI messages apply (when        sn_ols_flag of the scalable nesting SEI message is equal to 0)        or the lowest value of nuh_layer_id of all layers in the OLSs to        which the scalable-nested SEI message apply (when sn_ols_flag of        the scalable nesting SEI message is equal to 1).        -   NOTE 4—Same as for DCI, VPS, AUD, and EOB NAL units, the            value of nuh_layer_id for SEI NAL units that do not contain            a scalable nesting SEI message is not constrained.            It is a requirement of bitstream conformance that the            following restrictions apply on repetition of SEI messages:—    -   For each of the payloadType values included in        PicUnitRepConSeiList, there shall be less than or equal to 4        identical sei_payload( ) syntax structures within a PU.    -   There shall be less than or equal to 4 identical sei_payload( )        syntax structures with payloadType equal to 130 within a DU.        The following applies on the order of SLI, BP, PT, and DUI SEI        messages:    -   When an SLI SEI message and a BP SEI message that apply to a        particular OLS are present within an AU, the SLI SEI messages        shall precede the BP SEI message in decoding order.    -   When a BP SEI message and a PT SEI message that apply to a        particular OLS are present within an AU, the BP SEI messages        shall precede the PT SEI message in decoding order.    -   When a BP SEI message and a DUI SEI message that apply to a        particular OLS are present within an AU, the BP SEI messages        shall precede the DUI SEI message in decoding order.    -   When a PT SEI message and a DUI SEI message that apply to a        particular OLS are present within an AU, the PT SEI messages        shall precede the DUI SEI message in decoding order.

D.3.1 Buffering Period SEI Message Syntax

buffering_period( payloadSize ) { Descriptor  ... if(bp_max_sublayers_minus1 > 0 )  bp_sublayer_initial_cpb_removal_delay_present flag u(1 )  ... }

D.3.2 Buffering Period SEI Message Semantics

bp_sublayer_initial_cpb_removal_delay_present_flag equal to 1 specifiesthat initial CPB removal delay related syntax elements are present forsublayer representation(s) in the range of 0 to bp_max_sublayers_minus1,inclusive. bp_sublayer_initial_cpb_removal_delaypresent_flag equal to 0specifies that initial CPB removal delay related syntax elements arepresent for the bp_max_sublayers_minus1-th sublayer representation. Whennot present, the value ofbp_sublayer_initial_cpb_removal_delay_present_flag is inferred to beequal to 0.

FIG. 1 is a block diagram showing an example video processing system1900 in which various techniques disclosed herein may be implemented.Various implementations may include some or all of the components of thesystem 1900. The system 1900 may include input 1902 for receiving videocontent. The video content may be received in a raw or uncompressedformat, e.g., 8 or 10 bit multi-component pixel values, or may be in acompressed or encoded format. The input 1902 may represent a networkinterface, a peripheral bus interface, or a storage interface. Examplesof network interface include wired interfaces such as Ethernet, passiveoptical network (PON), etc. and wireless interfaces such as Wi-Fi orcellular interfaces.

The system 1900 may include a coding component 1904 that may implementthe various coding or encoding methods described in the presentdocument. The coding component 1904 may reduce the average bitrate ofvideo from the input 1902 to the output of the coding component 1904 toproduce a coded representation of the video. The coding techniques aretherefore sometimes called video compression or video transcodingtechniques. The output of the coding component 1904 may be eitherstored, or transmitted via a communication connected, as represented bythe component 1906. The stored or communicated bitstream (or coded)representation of the video received at the input 1902 may be used bythe component 1908 for generating pixel values or displayable video thatis sent to a display interface 1910. The process of generatinguser-viewable video from the bitstream representation is sometimescalled video decompression. Furthermore, while certain video processingoperations are referred to as “coding” operations or tools, it will beappreciated that the coding tools or operations are used at an encoderand corresponding decoding tools or operations that reverse the resultsof the coding will be performed by a decoder.

Examples of a peripheral bus interface or a display interface mayinclude universal serial bus (USB) or high definition multimediainterface (HDMI) or Displayport, and so on. Examples of storageinterfaces include SATA (serial advanced technology attachment), PCI,IDE interface, and the like. The techniques described in the presentdocument may be embodied in various electronic devices such as mobilephones, laptops, smartphones or other devices that are capable ofperforming digital data processing and/or video display.

FIG. 2 is a block diagram of a video processing apparatus 3600. Theapparatus 3600 may be used to implement one or more of the methodsdescribed herein. The apparatus 3600 may be embodied in a smartphone,tablet, computer, Internet of Things (IoT) receiver, and so on. Theapparatus 3600 may include one or more processors 3602, one or morememories 3604 and video processing hardware 3606. The processor(s) 3602may be configured to implement one or more methods described in thepresent document. The memory (memories) 3604 may be used for storingdata and code used for implementing the methods and techniques describedherein. The video processing hardware 3606 may be used to implement, inhardware circuitry, some techniques described in the present document.

FIG. 4 is a block diagram that illustrates an example video codingsystem 100 that may utilize the techniques of this disclosure.

As shown in FIG. 4 , video coding system 100 may include a source device110 and a destination device 120. Source device 110 generates encodedvideo data which may be referred to as a video encoding device.Destination device 120 may decode the encoded video data generated bysource device 110 which may be referred to as a video decoding device.

Source device 110 may include a video source 112, a video encoder 114,and an input/output (I/O) interface 116.

Video source 112 may include a source such as a video capture device, aninterface to receive video data from a video content provider, and/or acomputer graphics system for generating video data, or a combination ofsuch sources. The video data may comprise one or more pictures. Videoencoder 114 encodes the video data from video source 112 to generate abitstream. The bitstream may include a sequence of bits that form acoded representation of the video data. The bitstream may include codedpictures and associated data. The coded picture is a codedrepresentation of a picture. The associated data may include sequenceparameter sets, picture parameter sets, and other syntax structures. I/Ointerface 116 may include a modulator/demodulator (modem) and/or atransmitter. The encoded video data may be transmitted directly todestination device 120 via I/O interface 116 through network 130 a. Theencoded video data may also be stored onto a storage medium/server 130 bfor access by destination device 120.

Destination device 120 may include an I/O interface 126, a video decoder124, and a display device 122.

I/O interface 126 may include a receiver and/or a modem. I/O interface126 may acquire encoded video data from the source device 110 or thestorage medium/server 130 b. Video decoder 124 may decode the encodedvideo data. Display device 122 may display the decoded video data to auser. Display device 122 may be integrated with the destination device120, or may be external to destination device 120 which be configured tointerface with an external display device.

Video encoder 114 and video decoder 124 may operate according to a videocompression standard, such as the High Efficiency Video Coding (HEVC)standard, Versatile Video Coding (VVC) standard and other current and/orfurther standards.

FIG. 5 is a block diagram illustrating an example of video encoder 200,which may be video encoder 114 in the system 100 illustrated in FIG. 4 .

Video encoder 200 may be configured to perform any or all of thetechniques of this disclosure. In the example of FIG. 5 , video encoder200 includes a plurality of functional components. The techniquesdescribed in this disclosure may be shared among the various componentsof video encoder 200. In some examples, a processor may be configured toperform any or all of the techniques described in this disclosure.

The functional components of video encoder 200 may include a partitionunit 201, a predication unit 202 which may include a mode select unit203, a motion estimation unit 204, a motion compensation unit 205 and anintra prediction unit 206, a residual generation unit 207, a transformunit 208, a quantization unit 209, an inverse quantization unit 210, aninverse transform unit 211, a reconstruction unit 212, a buffer 213, andan entropy encoding unit 214.

In other examples, video encoder 200 may include more, fewer, ordifferent functional components. In an example, predication unit 202 mayinclude an intra block copy (IBC) unit. The IBC unit may performpredication in an IBC mode in which at least one reference picture is apicture where the current video block is located.

Furthermore, some components, such as motion estimation unit 204 andmotion compensation unit 205 may be highly integrated, but arerepresented in the example of FIG. 5 separately for purposes ofexplanation.

Partition unit 201 may partition a picture into one or more videoblocks. Video encoder 200 and video decoder 300 may support variousvideo block sizes.

Mode select unit 203 may select one of the coding modes, intra or inter,e.g., based on error results, and provide the resulting intra- orinter-coded block to a residual generation unit 207 to generate residualblock data and to a reconstruction unit 212 to reconstruct the encodedblock for use as a reference picture. In some example, Mode select unit203 may select a combination of intra and inter predication (CIIP) modein which the predication is based on an inter predication signal and anintra predication signal. Mode select unit 203 may also select aresolution for a motion vector (e.g., a sub-pixel or integer pixelprecision) for the block in the case of inter-predication.

To perform inter prediction on a current video block, motion estimationunit 204 may generate motion information for the current video block bycomparing one or more reference frames from buffer 213 to the currentvideo block. Motion compensation unit 205 may determine a predictedvideo block for the current video block based on the motion informationand decoded samples of pictures from buffer 213 other than the pictureassociated with the current video block.

Motion estimation unit 204 and motion compensation unit 205 may performdifferent operations for a current video block, for example, dependingon whether the current video block is in an I slice, a P slice, or a Bslice.

In some examples, motion estimation unit 204 may perform uni-directionalprediction for the current video block, and motion estimation unit 204may search reference pictures of list 0 or list 1 for a reference videoblock for the current video block. Motion estimation unit 204 may thengenerate a reference index that indicates the reference picture in list0 or list 1 that contains the reference video block and a motion vectorthat indicates a spatial displacement between the current video blockand the reference video block. Motion estimation unit 204 may output thereference index, a prediction direction indicator, and the motion vectoras the motion information of the current video block. Motioncompensation unit 205 may generate the predicted video block of thecurrent block based on the reference video block indicated by the motioninformation of the current video block.

In other examples, motion estimation unit 204 may perform bi-directionalprediction for the current video block, motion estimation unit 204 maysearch the reference pictures in list 0 for a reference video block forthe current video block and may also search the reference pictures inlist 1 for another reference video block for the current video block.Motion estimation unit 204 may then generate reference indexes thatindicate the reference pictures in list 0 and list 1 containing thereference video blocks and motion vectors that indicate spatialdisplacements between the reference video blocks and the current videoblock. Motion estimation unit 204 may output the reference indexes andthe motion vectors of the current video block as the motion informationof the current video block. Motion compensation unit 205 may generatethe predicted video block of the current video block based on thereference video blocks indicated by the motion information of thecurrent video block.

In some examples, motion estimation unit 204 may output a full set ofmotion information for decoding processing of a decoder.

In some examples, motion estimation unit 204 may do not output a fullset of motion information for the current video. Rather, motionestimation unit 204 may signal the motion information of the currentvideo block with reference to the motion information of another videoblock. For example, motion estimation unit 204 may determine that themotion information of the current video block is sufficiently similar tothe motion information of a neighboring video block.

In one example, motion estimation unit 204 may indicate, in a syntaxstructure associated with the current video block, a value thatindicates to the video decoder 300 that the current video block has thesame motion information as the another video block.

In another example, motion estimation unit 204 may identify, in a syntaxstructure associated with the current video block, another video blockand a motion vector difference (MVD). The motion vector differenceindicates a difference between the motion vector of the current videoblock and the motion vector of the indicated video block. The videodecoder 300 may use the motion vector of the indicated video block andthe motion vector difference to determine the motion vector of thecurrent video block.

As discussed above, video encoder 200 may predictively signal the motionvector. Two examples of predictive signaling techniques that may beimplemented by video encoder 200 include advanced motion vectorpredication (AMVP) and merge mode signaling.

Intra prediction unit 206 may perform intra prediction on the currentvideo block. When intra prediction unit 206 performs intra prediction onthe current video block, intra prediction unit 206 may generateprediction data for the current video block based on decoded samples ofother video blocks in the same picture. The prediction data for thecurrent video block may include a predicted video block and varioussyntax elements.

Residual generation unit 207 may generate residual data for the currentvideo block by subtracting (e.g., indicated by the minus sign) thepredicted video block(s) of the current video block from the currentvideo block. The residual data of the current video block may includeresidual video blocks that correspond to different sample components ofthe samples in the current video block.

In other examples, there may be no residual data for the current videoblock for the current video block, for example in a skip mode, andresidual generation unit 207 may not perform the subtracting operation.

Transform processing unit 208 may generate one or more transformcoefficient video blocks for the current video block by applying one ormore transforms to a residual video block associated with the currentvideo block.

After transform processing unit 208 generates a transform coefficientvideo block associated with the current video block, quantization unit209 may quantize the transform coefficient video block associated withthe current video block based on one or more quantization parameter (QP)values associated with the current video block.

Inverse quantization unit 210 and inverse transform unit 211 may applyinverse quantization and inverse transforms to the transform coefficientvideo block, respectively, to reconstruct a residual video block fromthe transform coefficient video block. Reconstruction unit 212 may addthe reconstructed residual video block to corresponding samples from oneor more predicted video blocks generated by the predication unit 202 toproduce a reconstructed video block associated with the current blockfor storage in the buffer 213.

After reconstruction unit 212 reconstructs the video block, loopfiltering operation may be performed reduce video blocking artifacts inthe video block.

Entropy encoding unit 214 may receive data from other functionalcomponents of the video encoder 200. When entropy encoding unit 214receives the data, entropy encoding unit 214 may perform one or moreentropy encoding operations to generate entropy encoded data and outputa bitstream that includes the entropy encoded data.

FIG. 6 is a block diagram illustrating an example of video decoder 300which may be video decoder 114 in the system 100 illustrated in FIG. 4 .

The video decoder 300 may be configured to perform any or all of thetechniques of this disclosure. In the example of FIG. 5 , the videodecoder 300 includes a plurality of functional components. Thetechniques described in this disclosure may be shared among the variouscomponents of the video decoder 300. In some examples, a processor maybe configured to perform any or all of the techniques described in thisdisclosure.

In the example of FIG. 6 , video decoder 300 includes an entropydecoding unit 301, a motion compensation unit 302, an intra predictionunit 303, an inverse quantization unit 304, an inverse transformationunit 305, and a reconstruction unit 306 and a buffer 307. Video decoder300 may, in some examples, perform a decoding pass generally reciprocalto the encoding pass described with respect to video encoder 200 (FIG. 5).

Entropy decoding unit 301 may retrieve an encoded bitstream. The encodedbitstream may include entropy coded video data (e.g., encoded blocks ofvideo data). Entropy decoding unit 301 may decode the entropy codedvideo data, and from the entropy decoded video data, motion compensationunit 302 may determine motion information including motion vectors,motion vector precision, reference picture list indexes, and othermotion information. Motion compensation unit 302 may, for example,determine such information by performing the AMVP and merge mode.

Motion compensation unit 302 may produce motion compensated blocks,possibly performing interpolation based on interpolation filters.Identifiers for interpolation filters to be used with sub-pixelprecision may be included in the syntax elements.

Motion compensation unit 302 may use interpolation filters as used byvideo encoder 20 during encoding of the video block to calculateinterpolated values for sub-integer pixels of a reference block. Motioncompensation unit 302 may determine the interpolation filters used byvideo encoder 200 according to received syntax information and use theinterpolation filters to produce predictive blocks.

Motion compensation unit 302 may uses some of the syntax information todetermine sizes of blocks used to encode frame(s) and/or slice(s) of theencoded video sequence, partition information that describes how eachmacroblock of a picture of the encoded video sequence is partitioned,modes indicating how each partition is encoded, one or more referenceframes (and reference frame lists) for each inter-encoded block, andother information to decode the encoded video sequence.

Intra prediction unit 303 may use intra prediction modes for examplereceived in the bitstream to form a prediction block from spatiallyadjacent blocks. Inverse quantization unit 303 inverse quantizes, i.e.,de-quantizes, the quantized video block coefficients provided in thebitstream and decoded by entropy decoding unit 301. Inverse transformunit 303 applies an inverse transform.

Reconstruction unit 306 may sum the residual blocks with thecorresponding prediction blocks generated by motion compensation unit202 or intra-prediction unit 303 to form decoded blocks. If desired, adeblocking filter may also be applied to filter the decoded blocks inorder to remove blockiness artifacts. The decoded video blocks are thenstored in buffer 307, which provides reference blocks for subsequentmotion compensation/intra predication and also produces decoded videofor presentation on a display device.

A listing of solutions describes some embodiments of the disclosedtechnology.

A first set of solutions is provided below. The following solutions showexample embodiments of techniques discussed in the previous section(e.g., items 1-3).

1. A video processing method (e.g., method 600 shown in FIG. 3 ),comprising: performing (602) a conversion between a video comprising oneor more video layers and a coded representation of the video comprisingone or more output layer sets, wherein the coded representation conformsto a format rule related to whether and how one or more syntax elementsrelated to a non-scalable-nested hypothetical reference decoder (HRD)related supplemental enhancement information (SEI).

2. The method of solution 1, wherein the format rule specifies that amessage related to the non-scalable-nested HRD related SEI applies tooutput layer sets that include same set of layers as the entire codedrepresentation.

3. The method of any of solutions 1-2, wherein the format rule specifiesto omit the one or more syntax elements in case that there is no outputlayer set that has a same set of layers as the entire codedrepresentation.

4. The method of solution 1, wherein the format rule specifies that theone or more syntax element are applicable to all layers in the codedrepresentation.

The following solutions show example embodiments of techniques discussedin the previous section (e.g., items 4-10).

5. A video processing method, comprising: performing a conversionbetween a video comprising one or more video layers and a codedrepresentation of the video comprising one or more output layer sets,wherein the coded representation conforms to a format rule related towhether and how one or more syntax elements are included in asupplemental enhancement information (SEI) network abstraction layer(NAL) unit.

6. The method of solution 5, wherein the format rule specifies that, incase that the SEI NAL unit includes a non-scalable-nested SEI message, avalue of layer identifier is unconstrained.

7. The method of any of solutions 5-6, wherein the format rule disablesinclusion of a SEI NAL unit that includes a scalable-nested SEI messageof a certain payload type due to signaling of use of a same picturetiming in all output layer sets of in the coded representation.

8. The method of any of solutions 5-7, wherein the format rule specifiesthat an SEI NAL unit that includes a non-scalable-nested SEI message ofa first specific payload type is disallowed to include another SEImessage of a second specific type.

9. The method of solution 8, wherein the first specific payload type isequal to 0, 1, 130 or 203.

10. The method of solutions 8 or 9, wherein the second specific payloadtype is equal to 0, 1, 130, 203, or 133.

11. The method of solutions 8 to 10, wherein first specific payload typeand the second specific payload type are 3.

12. The method of any of solutions 1-11, wherein the performing theconversion comprises encoding the video to generate the codedrepresentation.

13. The method of any of solutions 1-11, wherein the performing theconversion comprises parsing and decoding the coded representation togenerate the video.

14. A video decoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions 1 to 13.

15. A video encoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions 1 to 13.

16. A computer program product having computer code stored thereon, thecode, when executed by a processor, causes the processor to implement amethod recited in any of solutions 1 to 13.

17. A method, apparatus or system described in the present document.

A second set of solutions show example embodiments of techniquesdiscussed in the previous section (e.g., items 1-5).

1. A method of processing video data (e.g., method 700 as shown in FIG.7A), comprising: performing 702 a conversion between a video and abitstream of the video, wherein the bitstream includes one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that anon-scalable-nested supplemental enhancement information, SEI, messagethat includes information regarding hypothetical reference decoder, HRD,is applicable to all output layer sets that include same video layers asthe bitstream.

2. The method of solution 1, wherein the non-scalable-nested SEI messageis an SEI message that is not contained in a scalable nesting SEImessage.

3. The method of solution 1 or 2, wherein the non-scalable-nested SEImessage that includes information regarding the HRD is a bufferingperiod (BP) SEI message, a picture timing (PT) SEI message, a decodingunit information (DUI) SEI message, or a subpicture level information(SLI) SEI message.

4. The method of solution 1 or 2, wherein the non-scalable-nested SEImessage that includes information regarding the HRD has a payload typethat is equal to 0, 1, 130, or 203.

5. A method of processing video data (e.g., method 710 as shown in FIG.7B), comprising: performing 712 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that anon-scalable-nested hypothetical reference decoder (HRD)-relatedsupplemental enhancement information (SEI) message is omitted responsiveto a condition that there does not exist an output layer set thatincludes a same set of layers as the bitstream.

6. The method of solution 5, wherein the non-scalable-nested SEI messageis an SEI message that is not contained in a scalable nesting SEImessage.

7. The method of solution 5 or 6, wherein the non-scalable-nestedHRD-related SEI message is a buffering period (BP) SEI message, apicture timing (PT) SEI message, a decoding unit information (DUI) SEImessage, or a subpicture level information (SLI) SEI message.

8. The method of solution 5 or 6, wherein the non-scalable-nestedHRD-related SEI message has a payload type that is equal to 0, 1, 130,or 203.

9. A method of processing video data (e.g., method 720 as shown in FIG.7C), comprising: performing 722 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that a value of layeridentifier for a supplemental enhancement information (SEI) networkabstraction layer (NAL) unit that includes a non-scalable-nested SEImessage is not constrained.

10. The method of solution 9, wherein the non-scalable-nested SEImessage is an SEI message that is not contained in a scalable nestingSEI message.

11. The method of solution 9, wherein the non-scalable-nested SEImessage is a buffering period (BP) SEI message, a picture timing (PT)SEI message, a decoding unit information (DUI) SEI message, or asubpicture level information (SLI) SEI message.

12. The method of solution 9, wherein the non-scalable-nested SEImessage has a payload type that is equal to 0, 1, 130 or 203.

13. A method of processing video data (e.g., method 730 as shown in FIG.7D), comprising: performing 732 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to aformat rule, wherein the format rule specifies that a particular payloadtype value corresponding to subpicture level information is disallowedfrom a list that includes allowable supplemental enhancement information(SEI) payload type values for non-hypothetical reference decoder (HRD)related supplemental enhancement information (SEI) messages.

14. The method of solution 13, wherein the particular payload type valueis 203.

15. The method of solution 13, wherein the allowable SEI payload typevalues include filler payload, film grain characteristics, frame packingarrangement, parameter sets inclusion indication, mastering displaycolor volume, content light level information, dependent rap indication,alternative transfer characteristics, ambient viewing environment,content color volume, equirectangular projection, generalized cube mapprojection, sphere rotation, region-wise packing, omni viewport, framefield information and sample aspect ratio information.

16. The method of solution 13, wherein the allowable SEI payload typevalues include 3, 19, 45, 129, 137, 144, 145, 147 to 150, 153 to 156,168, and 204.

17. A method of processing video data, comprising: performing aconversion between a video and a bitstream of the video, wherein thebitstream includes one or more output layer sets comprising one or morevideo layers according to a format rule, wherein the format rulespecifies that a non-scalable-nested supplemental enhancementinformation, SEI, message that includes information irrelevant tohypothetical reference decoder (HRD) is applicable to all layers in thebitstream.

18. The method of solution 17, wherein the non-scalable-nested SEImessage is an SEI message that is not contained in a scalable nestingSEI message.

19. The method of solution 17 or 18, wherein the non-scalable-nested SEImessage that includes information irrelevant to the HRD has a payloadtype that is not equal to 0, 1, 130, or 203.

20. The method of solution 17 or 18, wherein the non-scalable-nested SEImessage that includes information irrelevant to the HRD does notcorrespond to a buffering period (BP) SEI message, a picture timing (PT)SEI message, a decoding unit information (DUI) SEI message, or asubpicture level information (SLI) SEI message.

21. The method of any of solutions 1 to 20, wherein the conversionincludes encoding the video into the bitstream.

22. The method of any of solutions 1 to 20, wherein the conversionincludes decoding the video from the bitstream.

23. The method of any of solutions 1 to 20, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

24. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 23.

25. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 23, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

26. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 23.

27. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

28. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions 1 to 23.

A third set of solutions show example embodiments of techniquesdiscussed in the previous section (e.g., item 6).

1. A method of processing video data (e.g., method 800 as shown in FIG.8 ), comprising: performing 802 a conversion between a video and abitstream of the video comprising one or more output layer setsaccording to a rule, wherein the rule specifies that a supplementalenhancement information (SEI) network abstraction layer (NAL) unit thatincludes a scalable-nested SEI message carrying picture timinginformation is not included due to use of a same picture timing in alloutput layer sets in the bitstream.

2. The method of solution 1, wherein the scalable-nested SEI message isan SEI message that is contained in a scalable nesting SEI message.

3. The method of solution 1 or 2, wherein the scalable-nested SEImessage carrying the picture timing information corresponds to a picturetiming (PT) SEI message.

4. The method of solution 1, wherein the scalable-nested SEI messagecarrying the picture timing information has a payload type equal to 1.

5. The method of any of solutions 1 to 4, wherein the rule furtherspecifies that non-scalable-nested SEI messages carrying the picturetiming information apply to the all output layer sets and subpicturesequences that are allowed to be extracted from the bitstream.

6. The method of solution 5, wherein a non-scalable-nested SEI messageis an SEI message that is not contained in a scalable nesting SEImessage.

7. The method of any of solutions 1 to 4, wherein the rule furtherspecifies that the SEI NAL unit that includes the scalable-nested SEImessage carrying the picture timing information is not included in casethat a syntax field has a value specifying that the scalable-nested SEImessage that applies to specific output layer sets or layers applies toall subpictures of the specified output layer sets or layers.

8. The method of any of solutions 1 to 7, wherein the conversionincludes encoding the video into the bitstream.

9. The method of any of solutions 1 to 7, wherein the conversionincludes decoding the video from the bitstream.

10. The method of any of solutions 1 to 7, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

11. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 10.

12. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 10, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

13. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 10.

14. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

15. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions 1 to 10.

A fourth set of solutions show example embodiments of techniquesdiscussed in the previous section (e.g., items 7-9).

1. A method of processing video data (e.g., method 900 as shown in FIG.9A), comprising: performing 902 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moresupplemental enhancement information, SEI, network abstraction layer(NAL) units according to a rule, wherein the rule specifies that,responsive to an SEI NAL unit including a non-scalable-nested SEImessage of a first payload type, the SEI NAL unit is disallowed toinclude another SEI message of a second payload type.

2. The method of solution 1, wherein the non-scalable-nested SEI messageis an SEI message that is not contained in a scalable nesting SEImessage.

3. The method of solution 1 or 2, wherein the non-scalable-nested SEImessage of the first payload type corresponds to a buffering period (BP)SEI message, a picture timing (PT) SEI message, a decoding unitinformation (DUI) SEI message, or a subpicture level information (SLI)SEI message.

4. The method of solution 1 or 2, wherein the first payload type isequal to 0, 1, 130 or 203.

5. The method of any of solutions 1 to 4, wherein thenon-scalable-nested SEI message of the second payload type does notcorrespond to any of a buffering period (BP) SEI message, a picturetiming (PT) SEI message, a decoding unit information (DUI) SEI message,and a subpicture level information (SLI) SEI message.

6. The method of any of solutions 1 to 4, wherein the second payloadtype is not equal to any of 0, 1, 130 and 203.

7. A method of processing video data (e.g., method 910 as shown in FIG.9B), comprising: performing 912 a conversion between a video and abitstream of the video according to a rule, wherein the bitstreamcomprises one or more supplemental enhancement information, SEI, networkabstraction layer (NAL) units according to a rule, wherein the rulespecifies that, responsive to an SEI NAL unit including ascalable-nested SEI message of a first payload type, the SEI NAL unit isdisallowed to include another SEI message of a second payload type.

8. The method of solution 7, wherein the scalable-nested SEI message isan SEI message that is contained in a scalable nesting SEI message.

9. The method of solution 7 or 8, wherein the scalable-nested SEImessage of the first payload type corresponds to a buffering period (BP)SEI message, a picture timing (PT) SEI message, a decoding unitinformation (DUI) SEI message, or a subpicture level information (SLI)SEI message.

10. The method of solution 7 or 8, wherein the first payload type isequal to 0, 1, 130 or 203.

11. The method of any of solutions 7 to 10, wherein the scalable-nestedSEI message of the second payload type does not correspond to any of abuffering period (BP) SEI message, a picture timing (PT) SEI message, adecoding unit information (DUI) SEI message, a subpicture levelinformation (SLI) SEI message, and a scalable nesting information SEImessage.

12. The method of any of solutions 7 to 10, wherein the second payloadtype is not equal to any of 0, 1, 130, 203, and 133.

13. A method of processing video data (e.g., method 920 as shown in FIG.9C), comprising: performing 922 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moresupplemental enhancement information, SEI, network abstraction layer(NAL) units according to a rule, wherein the rule specifies that,responsive to an SEI NAL unit including a SEI message of a first payloadtype, the SEI NAL unit is disallowed to include another SEI message notequal to the first payload type or a second payload type.

14. The method of solution 13, wherein the SEI message of the firstpayload type corresponds to a filler payload SEI message.

15. The method of solution 13 or 14, wherein the first payload type isequal to 3.

16. The method of any of solutions 13 to 15, wherein the rule furtherspecifies that the SEI message of the first payload type is disallowedto be contained in a scalable nesting SEI message.

17. The method of any of solutions 13 to 16, wherein the rule furtherspecifies that the SEI message of the second payload type is a scalablenesting SEI message.

18. The method of any of solutions 13 to 16, wherein the second payloadtype is equal to 133.

19. The method of any of solutions 1 to 18, wherein the conversionincludes encoding the video into the bitstream.

20. The method of any of solutions 1 to 18, wherein the conversionincludes decoding the video from the bitstream.

21. The method of any of solutions 1 to 18, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

22. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 21.

23. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 21, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

24. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 21.

25. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

26. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions 1 to 21.

A fifth set of solutions show example embodiments of techniquesdiscussed in the previous section (e.g., items 10 and 11).

1. A method of processing video data (e.g., method 1000 as shown in FIG.10A), comprising: performing 1002 a conversion between a video and abitstream of the video, wherein the bitstream comprises one or moreoutput layer sets comprising one or more video layers according to arule, wherein the rule specifies a particular decoding order between asubpicture level information (SLI) supplemental enhancement information(SEI) message and a buffering period (BP) SEI message that apply to aparticular output layer set, responsive to a condition that the SLI SEImessage and the BP SEI message are included in an access unit.

2. The method of solution 1, wherein the particular decoding order isthat the SLI SEI message precedes the BP SEI message.

3. A method of processing video data (e.g., method 1010 as shown in FIG.10B), comprising: performing 1012 a conversion between a video and abitstream of the video according to a rule, wherein the rule specifiesthat a first syntax field indicating sublayer representation informationfor which an initial coded picture buffer (CPB) removal delay relatedsyntax elements are present is omitted, responsive to a particular valueof a second syntax field indicative of a maximum number of temporalsublayers for which an initial CPB removal delay is indicated in abuffering period supplemental enhancement information (SEI) message.

4. The method of solution 3, wherein the particular value is 0.

5. The method of solution 3 or 4, wherein the second syntax filed withthe particular value specifies that the maximum number of temporalsublayers is 1.

6. The method of any of solutions 3 to 5, wherein the first syntax fieldequal to another particular value specifies that the initial CPB removaldelay related syntax elements are present for the sublayerrepresentation in a range of 0 to the second syntax field, inclusive.

7. The method of solution 6, wherein another particular value is 1.

8. The method of any of solutions 3 to 7, wherein the syntax filed equalto another particular value specifies that the initial CPB removal delayrelated syntax elements are present for the sublayer representationcorresponding to the particular value of the second syntax field.

9. The method of solution 8, wherein the another particular value is 0.

10. The method of any of solutions 3 to 9, wherein the rule furtherspecifies to infer a value of the first syntax field to be equal to 0,responsive to the particular value of the second syntax field.

11. The method of any of solutions 1 to 10, wherein the conversionincludes encoding the video into the bitstream.

12. The method of any of solutions 1 to 10, wherein the conversionincludes decoding the video from the bitstream.

13. The method of any of solutions 1 to 10, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

14. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 13.

15. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 13, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

16. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 13.

17. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

18. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions 1 to 13.

The disclosed and other solutions, examples, embodiments, modules andthe functional operations described in this document can be implementedin digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this document and theirstructural equivalents, or in combinations of one or more of them. Thedisclosed and other embodiments can be implemented as one or morecomputer program products, i.e., one or more modules of computer programinstructions encoded on a computer readable medium for execution by, orto control the operation of, data processing apparatus. The computerreadable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this document can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random-access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto optical disks; and CD ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any subject matter or of whatmay be claimed, but rather as descriptions of features that may bespecific to particular embodiments of particular techniques. Certainfeatures that are described in this patent document in the context ofseparate embodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

1. A method of processing video data, comprising: performing aconversion between a video and a bitstream of the video, wherein thebitstream comprises one or more output layer sets comprising one or morevideo layers according to a format rule, wherein the format rulespecifies that a value of layer identifier for a supplementalenhancement information (SEI) network abstraction layer (NAL) unit thatincludes a non-scalable-nested SEI message is not constrained.
 2. Themethod of claim 1, wherein the non-scalable-nested SEI message is an SEImessage that is not contained in a scalable nesting SEI message.
 3. Themethod of claim 1, wherein the format rule further specifies that anon-scalable-nested SEI message that includes information regardinghypothetical reference decoder (HRD) is applicable to all output layersets that include video layers as the bitstream.
 4. The method of claim3, wherein the format rule further specifies that a non-scalable-nestedhypothetical reference decoder (HRD)-related supplemental enhancementinformation (SEI) message is omitted responsive to a condition thatthere does not exist an output layer set that includes the video layersas the bitstream.
 5. The method of claim 4, wherein thenon-scalable-nested SEI message is a buffering period (BP) SEI message,a picture timing (PT) SEI message, a decoding unit information (DUI) SEImessage, or a subpicture level information (SLI) SEI message.
 6. Themethod of claim 5, wherein the non-scalable-nested SEI message has apayload type that is equal to 0, 1, 130 or
 203. 7. The method of claim6, wherein the format rule further specifies that a particular payloadtype value corresponding to subpicture level information is disallowedfrom a list that includes allowable supplemental enhancement information(SEI) payload type values.
 8. The method of claim 7, wherein theparticular payload type value is
 203. 9. The method of claim 8, whereinthe allowable SEI payload type values include filler payload, film graincharacteristics, frame packing arrangement, parameter sets inclusionindication, mastering display color volume, content light levelinformation, dependent rap indication, alternative transfercharacteristics, ambient viewing environment, content color volume,equirectangular projection, generalized cube map projection, sphererotation, region-wise packing, omni viewport, frame field informationand sample aspect ratio information.
 10. The method of claim 9, whereinthe allowable SEI payload type values include 3, 19, 45, 129, 137, 144,145, 147 to 150, 153 to 156, 168, and
 204. 11. The method of claim 9,wherein the SEI messages corresponding the allowable SEI payload typevalues of the list infer constraints on NAL unit header of the SEI NALunit on the basis of the NAL unit header of associated video codinglayer (VCL) NAL unit when non-scalable-nested.
 12. The method of claim1, wherein the conversion includes encoding the video into thebitstream.
 13. The method of claim 1, wherein the conversion includesdecoding the video from the bitstream.
 14. An apparatus for processingvideo data comprising a processor and a non-transitory memory withinstructions thereon, wherein the instructions upon execution by theprocessor, cause the processor to: perform a conversion between a videoand a bitstream of the video, wherein the bitstream comprises one ormore output layer sets comprising one or more video layers according toa format rule, wherein the format rule specifies that a value of layeridentifier for a supplemental enhancement information (SEI) networkabstraction layer (NAL) unit that includes a non-scalable-nested SEImessage is not constrained.
 15. The apparatus of claim 14, wherein thenon-scalable-nested SEI message is an SEI message that is not containedin a scalable nesting SEI message.
 16. The apparatus of claim 14,wherein the format rule further specifies that the format rule furtherspecifies that a non-scalable-nested SEI message that includesinformation regarding hypothetical reference decoder (HRD) is applicableto all output layer sets that include video layers as the bitstream,wherein the format rule further specifies that a non-scalable-nestedhypothetical reference decoder (HRD)-related supplemental enhancementinformation (SEI) message is omitted responsive to a condition thatthere does not exist an output layer set that includes the video layersas the bitstream, wherein the non-scalable-nested SEI message is abuffering period (BP) SEI message, a picture timing (PT) SEI message, adecoding unit information (DUI) SEI message, or a subpicture levelinformation (SLI) SEI message, wherein the non-scalable-nested SEImessage has a payload type that is equal to 0, 1, 130 or 203, whereinthe format rule further specifies that a particular payload type valuecorresponding to subpicture level information is disallowed from a listthat includes allowable supplemental enhancement information (SEI)payload type values, wherein the particular payload type value is 203,wherein the allowable SEI payload type values include filler payload,film grain characteristics, frame packing arrangement, parameter setsinclusion indication, mastering display color volume, content lightlevel information, dependent rap indication, alternative transfercharacteristics, ambient viewing environment, content color volume,equirectangular projection, generalized cube map projection, sphererotation, region-wise packing, omni viewport, frame field informationand sample aspect ratio information.
 17. A non-transitorycomputer-readable storage medium storing instructions that cause aprocessor to: perform a conversion between a video and a bitstream ofthe video, wherein the bitstream comprises one or more output layer setscomprising one or more video layers according to a format rule, whereinthe format rule specifies that a value of layer identifier for asupplemental enhancement information (SEI) network abstraction layer(NAL) unit that includes a non-scalable-nested SEI message is notconstrained.
 18. The non-transitory computer-readable storage medium ofclaim 17, wherein the non-scalable-nested SEI message is an SEI messagethat is not contained in a scalable nesting SEI message; wherein theformat rule further specifies that a non-scalable-nested SEI messagethat includes information regarding hypothetical reference decoder (HRD)is applicable to all output layer sets that include video layers as thebitstream, wherein the format rule further specifies that anon-scalable-nested hypothetical reference decoder (HRD)-relatedsupplemental enhancement information (SEI) message is omitted responsiveto a condition that there does not exist an output layer set thatincludes the video layers as the bitstream, wherein thenon-scalable-nested SEI message is a buffering period (BP) SEI message,a picture timing (PT) SEI message, a decoding unit information (DUI) SEImessage, or a subpicture level information (SLI) SEI message, whereinthe non-scalable-nested SEI message has a payload type that is equal to0, 1, 130 or 203, wherein the format rule further specifies that aparticular payload type value corresponding to subpicture levelinformation is disallowed from a list that includes allowablesupplemental enhancement information (SEI) payload type values, whereinthe particular payload type value is 203, wherein the allowable SEIpayload type values include filler payload, film grain characteristics,frame packing arrangement, parameter sets inclusion indication,mastering display color volume, content light level information,dependent rap indication, alternative transfer characteristics, ambientviewing environment, content color volume, equirectangular projection,generalized cube map projection, sphere rotation, region-wise packing,omni viewport, frame field information and sample aspect ratioinformation.
 19. A non-transitory computer-readable recording mediumstoring a bitstream of a video which is generated by a method performedby a video processing apparatus, wherein the method comprises:generating the bitstream of the video, wherein the bitstream comprisesone or more output layer sets comprising one or more video layersaccording to a format rule, wherein the format rule specifies that avalue of layer identifier for a supplemental enhancement information(SEI) network abstraction layer (NAL) unit that includes anon-scalable-nested SEI message is not constrained.
 20. Thenon-transitory computer-readable recording medium of claim 19, whereinthe non-scalable-nested SEI message is an SEI message that is notcontained in a scalable nesting SEI message; wherein the format rulefurther specifies that a non-scalable-nested SEI message that includesinformation regarding hypothetical reference decoder (HRD) is applicableto all output layer sets that include video layers as the bitstream,wherein the format rule further specifies that a non-scalable-nestedhypothetical reference decoder (HRD)-related supplemental enhancementinformation (SEI) message is omitted responsive to a condition thatthere does not exist an output layer set that includes the video layersas the bitstream, wherein the non-scalable-nested SEI message is abuffering period (BP) SEI message, a picture timing (PT) SEI message, adecoding unit information (DUI) SEI message, or a subpicture levelinformation (SLI) SEI message, wherein the non-scalable-nested SEImessage has a payload type that is equal to 0, 1, 130 or 203, whereinthe format rule further specifies that a particular payload type valuecorresponding to subpicture level information is disallowed from a listthat includes allowable supplemental enhancement information (SEI)payload type values, wherein the particular payload type value is 203,wherein the allowable SEI payload type values include filler payload,film grain characteristics, frame packing arrangement, parameter setsinclusion indication, mastering display color volume, content lightlevel information, dependent rap indication, alternative transfercharacteristics, ambient viewing environment, content color volume,equirectangular projection, generalized cube map projection, sphererotation, region-wise packing, omni viewport, frame field informationand sample aspect ratio information.